# Efficient Pretraining

GenMedClip is a zero-shot image classification model based on the open_clip library, specializing in medical image analysis.

Rho-1 is a language model specialized in mathematics, pretrained with Selective Language Modeling (SLM) method, significantly improving accuracy in solving mathematical problems.

TinyLlama is a compact language model with 1.1 billion parameters, adopting the same architecture and tokenizer as Llama 2, suitable for applications with limited computational and memory resources.

TinyLlama is a 1.1B parameter Llama model pretrained on 3 trillion tokens, designed to provide compact and efficient text generation capabilities.

TinyLlama is a compact 1.1B-parameter Llama model pretrained on 3 trillion tokens, designed for resource-constrained environments.

Sheared-LLaMA-2.7B is a lightweight language model derived from Llama-2-7b through pruning and continued pretraining, consuming only a 50B token budget.

TinyLlama is a 1.1B parameter Llama model, planned to be pretrained on 3 trillion tokens, optimized to complete training in 90 days on 16 A100-40G GPUs.

CodeT5+ 16B is an open-source family of large language models for code, featuring an encoder-decoder architecture that supports multiple modes, suitable for a wide range of code understanding and generation tasks.

VideoMAE is a masked autoencoder model for video, pretrained with self-supervision and fine-tuned on the Kinetics-400 dataset, suitable for video classification tasks.

VideoMAE is a video pretraining model based on Masked Autoencoder (MAE), fine-tuned on the Kinetics-400 dataset through self-supervised learning, suitable for video classification tasks.

The first pre-trained language model specifically for Indonesian Twitter, built by extending Indonesian BERT with domain-specific vocabulary

An efficient Arabic language model based on Funnel Transformer and ELECTRA objective, with low computational cost and superior performance

BERTIN is a series of Spanish BERT-based models. The current model is a RoBERTa-base model trained from scratch on a portion of the Spanish mC4 dataset using Flax.

This is a sparsely pretrained BERT-Base model achieving 90% weight sparsity through one-shot pruning, suitable for fine-tuning on various language tasks.